[IGN-13784, IGN-13799] AD and Postgres issues on 8.3 Upgrade

With a standard upgrade we found these initial issues:

AD User source added ""s in the Search Base, Role Search Base fields. Nothing resolved until the "s were removed. (Solved)

Postgres connection using Client Cert auth:

Cannot create PoolableConnectionFactory (Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8".)

Params:

"?ssl=true;sslmode=verify-full;sslrootcert=/etc/step/certs/root_ca.crt;sslcert=/etc/step/certs/ssl-cert.pem;sslkey=/etc/step/certs/ssl-cert.key.pk8"

1 Like

Did you verify this file exists at that location? Check permission? What user is Ignition running as, what users have read permission?

Were these previously empty and they became "" instead?

The fields before upgrade were not empty. They had functional and correct values. It seems the upgrade itself added the quote marks.

Yes the file is still present. The redundant gateway (we only upgraded the primary to see what would happen) continues to successfully access the file and have a good db connection.

Is there a full stack trace to go with this? Can you upload log files?

java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8".)
	at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:645)
	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)
	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8".
	at org.postgresql.ssl.LazyKeyManager.getPrivateKey(LazyKeyManager.java:268)
	at java.base/sun.security.ssl.AbstractKeyManagerWrapper.getPrivateKey(Unknown Source)
	at java.base/sun.security.ssl.X509Authentication.createClientPossession(Unknown Source)
	at java.base/sun.security.ssl.X509Authentication.createPossession(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.choosePossession(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
	at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Unknown Source)
	at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:465)
	at org.postgresql.Driver.connect(Driver.java:264)
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:481)
	at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:114)
	at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:641)
	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)
	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /etc/step/certs/ssl-cert.key.pk8" (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(Unknown Source)
	at java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	at java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	at java.base/java.io.RandomAccessFile.<init>(Unknown Source)
	at org.postgresql.ssl.LazyKeyManager.readFileFully(LazyKeyManager.java:182)
	at org.postgresql.ssl.LazyKeyManager.getPrivateKey(LazyKeyManager.java:205)
	at java.base/sun.security.ssl.AbstractKeyManagerWrapper.getPrivateKey(Unknown Source)
	at java.base/sun.security.ssl.X509Authentication.createClientPossession(Unknown Source)
	at java.base/sun.security.ssl.X509Authentication.createPossession(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.choosePossession(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(Unknown Source)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
	at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Unknown Source)
	at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:465)
	at org.postgresql.Driver.connect(Driver.java:264)
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:481)
	at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:114)
	at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:641)
	at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)
	at com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)
	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

I assure you that the file exists. The service is running as root (I know, but moving on) and there are no issues with root accessing the file or directory. This was working perfectly before the upgrade. My connection parameters are:

"?ssl=true;sslmode=verify-full;sslrootcert=/etc/step/certs/root_ca.crt;sslcert=/etc/step/certs/ssl-cert.pem;sslkey=/etc/step/certs/ssl-cert.key.pk8"

Note the quotation marks around the whole thing. Those weren't there initially when this was imported and I had to add them to even get this far.

Huh, that's interesting. And there's another post where somebody's AD config gained some quotation marks that they had to remove...

Looks like the quote is part of the filename it's attempting to use.

Same somebodys Jay and I work together. We were complaining about that up this thread.

2 Likes

oh duh, that was this thread :man_facepalming:

2 Likes

That's a good point. I've tried adding a space after the file name and it just considers the space and quote to be part of the filename. Without the quotes I get an error, Cannot create PoolableConnectionFactory (The server requested password-based authentication, but no password was provided.) so Ignition isn't even trying to do client cert authentication.

I've tried various combinations of single quotes double quotes, escaping the quotes with \ in front of them etc.

Here's the output of the get database connection config API endpoint if it provides any insight into how Ignition is parsing things:

{
  "type": "ignition/database-connection",
  "name": "qtspgc1after83",
  "description": "",
  "enabled": true,
  "version": 1,
  "collection": "core",
  "collections": [
    "core"
  ],
  "signature": "564d3ef93ec7c8772088021363a3ec2492481b3a8eb4fbe8622cbb4859967ccd",
  "config": {
    "driver": "PostgreSQL",
    "translator": "POSTGRES",
    "includeSchemaInTableName": false,
    "connectURL": "jdbc:postgresql://qts-pgc1cf.atlashq.com:5432/testdb",
    "username": "ig_atl",
    "connectionProps": "\"?ssl=true;sslmode=verify-full;sslrootcert=/etc/step/certs/root_ca.crt;sslcert=/etc/step/certs/ssl-cert.pem;sslkey=/etc/step/certs/ssl-cert.key.pk8\"",
    "connectionResetParams": "",
    "defaultTransactionLevel": "DEFAULT",
    "poolInitSize": 0,
    "poolMaxActive": 8,
    "poolMaxIdle": 8,
    "poolMinIdle": 0,
    "poolMaxWait": 5000,
    "validationQuery": "SELECT 1",
    "testOnBorrow": true,
    "testOnReturn": false,
    "testWhileIdle": false,
    "evictionRate": -1,
    "evictionTests": 3,
    "evictionTime": 1800000,
    "failoverProfile": "",
    "failoverMode": "STANDARD",
    "slowQueryLogThreshold": 60000,
    "validationSleepTime": 10000
  },
  "data": [
    "config.json"
  ],
  "attributes": {
    "lastModification": {
      "actor": "mquinn",
      "timestamp": "2025-07-29T19:27:14Z"
    },
    "uuid": "c0cc80ea-a98d-436e-87c1-df11eebeede8",
    "lastModificationSignature": "72cbcef1201a27eb7a4eb89501fd0ba3f2ac50ccee793d922200a4ff8ca82393",
    "enabled": true
  },
  "metrics": {
    "active-connections": {
      "name": "databases.connection-qtspgc1after83.active-connections",
      "metric": {
        "value": 0.0,
        "numerator": 0.0,
        "denominator": 8.0,
        "type": "ratio-gauge",
        "description": "Active Connections",
        "units": ""
      }
    },
    "active-queries": {
      "name": "databases.connection-qtspgc1after83.active-queries",
      "metric": {
        "value": [],
        "type": "gauge",
        "description": "Active Queries",
        "units": "queries"
      }
    },
    "expensive-queries": {
      "name": "databases.connection-qtspgc1after83.expensive-queries",
      "metric": {
        "value": [],
        "type": "gauge",
        "description": "Expensive Queries",
        "units": "queries"
      }
    },
    "queries": {
      "name": "databases.connection-qtspgc1after83.queries",
      "metric": {
        "oneMinuteRate": 0.0,
        "fiveMinuteRate": 0.0,
        "fifteenMinuteRate": 0.0,
        "count": 0,
        "meanRate": 0.0,
        "histogram": {
          "max": 0.0,
          "min": 0.0,
          "p50": 0.0,
          "p75": 0.0,
          "p95": 0.0,
          "p98": 0.0,
          "p99": 0.0,
          "mean": 0.0,
          "bucketSize": 0.0,
          "buckets": [
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0
          ],
          "type": "histogram",
          "stdev": 0.0
        },
        "type": "timer"
      }
    },
    "rows": {
      "name": "databases.connection-qtspgc1after83.rows",
      "metric": {
        "oneMinuteRate": 0.0,
        "fiveMinuteRate": 0.0,
        "fifteenMinuteRate": 0.0,
        "count": 0,
        "meanRate": 0.0,
        "type": "meter"
      }
    },
    "throughput": {
      "name": "databases.connection-qtspgc1after83.throughput",
      "metric": {
        "value": -1.0,
        "type": "gauge",
        "description": "",
        "units": "queries/sec"
      }
    }
  },
  "healthchecks": {
    "status": {
      "name": "databases.connection-qtspgc1after83.status",
      "result": {
        "healthy": false,
        "time": 1753817303275,
        "message": "Faulted",
        "error": {
          "message": "Cannot create PoolableConnectionFactory (Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8\".)",
          "stacktrace": [
            "java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8\".)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:645)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)",
            "\tat com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)",
            "\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)",
            "\tat java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)",
            "\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
            "\tat java.base/java.lang.Thread.run(Unknown Source)",
            "Caused by: org.postgresql.util.PSQLException: Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8\".",
            "\tat org.postgresql.ssl.LazyKeyManager.getPrivateKey(LazyKeyManager.java:268)",
            "\tat java.base/sun.security.ssl.AbstractKeyManagerWrapper.getPrivateKey(Unknown Source)",
            "\tat java.base/sun.security.ssl.X509Authentication.createClientPossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.X509Authentication.createPossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.choosePossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)",
            "\tat java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Unknown Source)",
            "\tat java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)",
            "\tat java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)",
            "\tat org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)",
            "\tat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)",
            "\tat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)",
            "\tat org.postgresql.Driver.makeConnection(Driver.java:465)",
            "\tat org.postgresql.Driver.connect(Driver.java:264)",
            "\tat org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)",
            "\tat org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:481)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:114)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:641)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)",
            "\tat com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)",
            "\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)",
            "\tat java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)",
            "\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
            "\tat java.base/java.lang.Thread.run(Unknown Source)",
            "Caused by: java.io.FileNotFoundException: /etc/step/certs/ssl-cert.key.pk8\" (No such file or directory)",
            "\tat java.base/java.io.RandomAccessFile.open0(Native Method)",
            "\tat java.base/java.io.RandomAccessFile.open(Unknown Source)",
            "\tat java.base/java.io.RandomAccessFile.<init>(Unknown Source)",
            "\tat java.base/java.io.RandomAccessFile.<init>(Unknown Source)",
            "\tat java.base/java.io.RandomAccessFile.<init>(Unknown Source)",
            "\tat org.postgresql.ssl.LazyKeyManager.readFileFully(LazyKeyManager.java:182)",
            "\tat org.postgresql.ssl.LazyKeyManager.getPrivateKey(LazyKeyManager.java:205)",
            "\tat java.base/sun.security.ssl.AbstractKeyManagerWrapper.getPrivateKey(Unknown Source)",
            "\tat java.base/sun.security.ssl.X509Authentication.createClientPossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.X509Authentication.createPossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.choosePossession(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(Unknown Source)",
            "\tat java.base/sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)",
            "\tat java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Unknown Source)",
            "\tat java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)",
            "\tat java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)",
            "\tat java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)",
            "\tat org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)",
            "\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)",
            "\tat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)",
            "\tat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)",
            "\tat org.postgresql.Driver.makeConnection(Driver.java:465)",
            "\tat org.postgresql.Driver.connect(Driver.java:264)",
            "\tat org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)",
            "\tat org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:481)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:114)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:641)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:546)",
            "\tat org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:723)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.getConnectionInternal(DatasourceImpl.java:360)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceImpl.runTest(DatasourceImpl.java:312)",
            "\tat com.inductiveautomation.ignition.gateway.datasource.DatasourceManagerImpl$FaultedDatasourceRetryer.lambda$newRetryRunnable$0(DatasourceManagerImpl.java:1109)",
            "\tat com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:593)",
            "\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)",
            "\tat java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)",
            "\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
            "\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
            "\tat java.base/java.lang.Thread.run(Unknown Source)"
          ]
        }
      }
    }
  }
}

Hi - I don't suppose you have a gateway backup that we could get that demonstrates the LDAP issue? I have tried to reproduce the issue, but am not having any luck.

Even screenshots (or copy/paste) of the values (pre and post migration), if you can share them... would be great.

You should have a DM with the gwbks.

These were running on Ubuntu 24.04 and upgraded from 8.1 using the zip installer and the instructions in the README.

I was able to reproduce with the gateway backup you sent me.

Your 8.1 configuration contained a role and user search base that both contained double-apostrophes surrounding the search base, and these apostrophes were migrated to 8.3. The apostrophes didn't appear to affect JNDI in 8.1, which I am quite surprised.

However; in 8.3 we have now added some code during the work for nested group membership that requires these to be valid distinguished names. If the search bases exist, we use them in a few ways:

  • As the base entry for the JNDI search (existed in 8.1)
  • In group membership lookups to verify that referred users and groups have distinguished names that fall under the listed base entry for the entity type (user/role).

Obviously, the distinguished names returned from the LDAP server will not have the apostrophes and therefore we need to make sure the search bases are also valid distinguished names so we can do the comparison. Groups and users that are returned that do not fall under the listed search bases are ignored.

So unfortunately, you will need to remove the apostrophes in 8.1, or in 8.3 after migration. We intend to add a note to the upgrade guide to help others who might run into the same issue.

2 Likes

Hi Jesse,
I had a similar issue while upgrading my test environment from 8.1 to 8.3-beta. The autentication with the AD user source worked well on 8.1 but in 8.3 the "User Role Name Attribute" had been changed from "cn" to "sAMAccountName". That leads to the problem that the user roles cannot be fetched anymore. The even more strange thing is, that I cannot change the "User Role Name Attribute" back to "cn" as this will automatically change the "User Name Attribute" field. In our case that results in the sitatuation that we are not able to use AD user sources anymore.


1 Like

BluegelMarius -

I will take a look at this. If you have a gateway backup that you are willing to send me, that might expedite the process.

BluegelMarius -

I took a quick look at the edit AD panel in the Web UI, and there is a bug where it maps both the user name attribute and the user role name attribute to the same field. I will open a ticket for this issue.

In the mean time, you can use the OpenAPI doc to learn how to send the command to update the user source directly, with your favorite HTTP client:

  • http://<YOUR_HOST>:<YOUR_PORT>/openapi

You need to be authenticated to view the OpenAPI doc. You can search for modify user sources or navigate to config user source and expand it to find the modify doc.

You will need to generate an API token if you use a separate HTTP client, or if you have an HTTP client that can run in the web browser, it should be able to make use of the session/cookies you already have from logging in.

To help get a properly formatted JSON payload for this command, the easiest way is to issue the command through the Web UI and copy that payload, and modify it accordingly. In this case config.settings.userNameAttribute and config.settings.roleNameAttribute.

Let me know if that doesn't work.

Jesse

EDITED:

I should also ask, what was the state of the settings for the AD user source immediately after migration? Due to the WebUI bug, you would need to look at the config directly, by doing one of the following:

  • Use the OpenAPI doc mentioned in my previous comment.
  • Look at the payload when you load the edit user source panel in the WebUI using developer tools.
  • Look at the user resource on disk (home/data/config/resources/core/ignition/user-source/<PROFILE_NAME>/config.json)

Do the settings look right using one of those methods immediately after doing a migration?

4 Likes