New Install, observing "trustAnchors parameter must be non-empty" in the error logs

Ignition
1

Greetings!

I have a new installation of a 8.1.16 Gateway running on RHEL 8.5 latest.

We are using Lets Encrypt certificates pulled via Hashicorp Vault Agent, that basically triggers the script referenced in this wonderful blog post.

The Gateway works/starts and is using the certificate desired. However after peeking at the logs a bit I’m noticing the following:

INFO   | jvm 1    | 2022/05/02 16:17:27 | java.lang.Exception: PKIX certificate path validation failed
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:292)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:109)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.SslManager.validateKeyStore(SslManager.java:261)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.SslManager$AbstractStateReader.readState(SslManager.java:301)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.SslManager.refreshInternal(SslManager.java:423)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.SslManager.refresh(SslManager.java:403)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.SslManager.refresh(SslManager.java:414)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.GcuInterface.reloadSslKeyStore(GcuInterface.java:444)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.GcuInterface.requestReceieved(GcuInterface.java:493)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.catapult.GcuRequestServer.handleRequest(GcuRequestServer.java:60)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.catapult.GcuRequestServer$RequestFileMonitor$2.run(GcuRequestServer.java:118)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.security.cert.PKIXParameters.setTrustAnchors(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.security.cert.PKIXParameters.<init>(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at java.base/java.security.cert.PKIXBuilderParameters.<init>(Unknown Source)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	at com.inductiveautomation.ignition.gateway.ssl.CertificateValidationUtil.verifyTrustChain(CertificateValidationUtil.java:230)
INFO   | jvm 1    | 2022/05/02 16:17:27 | 	... 13 common frames omitted
INFO   | jvm 1    | 2022/05/02 16:17:27 | I [g.SslManager                  ] [21:17:27]: State refreshed state=CA_SIGNED_CERTIFICATE
INFO   | jvm 1    | 2022/05/02 16:17:27 | I [o.e.j.u.s.SslContextFactory   ] [21:17:27]: x509=X509@4ddf6087(ignition,h=[mydomain.com.com],a=[],w=[mydomain.com, linux.mydomain.com]) for Server@37c72a14[provider=null,keyStore=null,trustStore=null]

Browsing a bit online, it seem to reference an incorrect/missing/wrong certificate chain. I believe I have all the certificates in the chain…

 0: subject=CN = *.mydomain.com
issuer=C = US, O = Let's Encrypt, CN = R3
 1: subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2: subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

As stated, everything appears to be functional I’m just a bit perplex why I’m seeing the following in the logs. Anyone seen anything similar?

Thanks!
~Jordan

1 Like
2

How often is this message appearing in the logs? If everything is functional, are you able to connect to the gateway with clients over SSL?

3

Message in the log shows up consistently every 5 minutes. However the gateway works and functions as expected over SSL. Just very noisy in the logs.

Thanks!

4

Upgraded to Ignition 8.1.18, thinking maybe something would change and nope! Still generating every 5 minutes.

5

You’re missing a certificate in the chain. Entry 2 says the issuer is “Digital Signature Trust Co., CN = DST Root CA X3”.

A complete chain would have that as entry 3, pointing to itself as issuer because it’s a root/CA.

edit: alternatively, this root/CA cert is not part of the default cacerts for some reason and needs to be added via the supplemental certificate mechanism: Adding Security Certificates into KeyStores - Ignition User Manual 8.1 - Ignition Documentation. Or maybe both?

6

I thought the version of java can determine what comes preloaded in the keystore. This isn’t a random or self owned CA. I know when LE updated their intermediate/root CA it broke certain older versions of java. This was last year I think?

All the “new stuff” came with this so batteries included. The CA exists in the OS’s ca-certifcate store but Java keystore is always another special layer…

Is there a default password for the java keystore that’s provided with Ignition? It seems to want a password and “ignition” doesn’t seem to suffice. I wanted to dump the existing keystore to see whats all included (if anything).

# ./keytool -list -v -keystore ../lib/security/cacerts
Warning: use -cacerts option to access cacerts keystore
Enter keystore password:  
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
java.io.IOException: Keystore was tampered with, or password was incorrect
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source)
	at java.base/java.security.KeyStore.load(Unknown Source)
	at java.base/sun.security.tools.keytool.Main.doCommands(Unknown Source)
	at java.base/sun.security.tools.keytool.Main.run(Unknown Source)
	at java.base/sun.security.tools.keytool.Main.main(Unknown Source)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
7

The default password is the same as it is for any standard JDK: changeit

8

I’m not sure it matters if that root CA is in the cacerts or not.

I’m looking at the code path in the stack trace and it’s trying to validate the certificates you’ve put into the SSL KeyStore, and we expect the full chain to be in the keystore, not the chain minus the root CA, regardless of whether that root CA is one of the default trusted CAs or not.

9

It looks like the consequence of this is that it will seem to work, as you’re seeing, but will eventually fail to pick up any changes that happen to the KeyStore until you restart the gateway, because it’s the refresh mechanism doing the validation that is failing.

10

So it is expecting the full chain, regardless if it already is in the java cacertificates?

I did dump the keystore and looked. Thanks to you for the password! ISRG Root X1 is certainly already included.

11

Yes, it’s expected.

I’m talking this over right now with a colleague and it’s looking like you can just ignore this warning for now. We’re talking about ways we can fix this without requiring the root to be part of the chain, but if you want to fix the warning for now then you can add it.

12

Thank you once again Kevin!

I sometimes get split brain because of how Java hands certs vs something like the normal OS cacert store. As long as the root is already in place (not a custom CA) then generally you are good to go. If anything is from a custom CA, then most certainly need to make sure they are added.

The certificate was working just fine, I was mostly interested in cleaning up the logs a bit since we will be pushing those to a central repository.

Again, thank you!

13

It's not just Java these days, browsers are moving to this model as well.

14

Good to know - I will mark this solved. Thanks again so much!