Resolving Ignition 8.3 Upgrade AD and Postgres mTLS Failures

This gateway was upgraded from 8.1 to 8.3 with the zip installer on Ubuntu 24.04. Only the primary node of the redundant pair was upgraded. After the upgrade, its Active Directory user source resolved no users or roles. Its PostgreSQL connection also faulted with Cannot create PoolableConnectionFactory (Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8".). The backup node stayed on 8.1 and kept a healthy database connection using the same certificate path.

Three separate defects stack up in this failure. They are tracked under IGN-13784, IGN-13799, IGN-13846 and IGN-14121:

• Quoted AD search bases that 8.3 no longer tolerates.
• An AD edit-panel field-mapping bug.
• A change in how 8.3 passes the database username to the JDBC driver when the password field is empty.

Which hop fails for each post-upgrade symptom?

Sort the symptoms by where the request stops before you change any settings:

• The AD faults stop inside the gateway, after the LDAP server has already answered.
• The Postgres faults stop in one of two places: the gateway's own TLS client, or the server's authentication stage after TLS completes.

Symptom	Hop where the request stops	Root cause	Resolution / status
AD user source resolves no users or roles; nothing works until quote characters are removed	Gateway-side filtering of LDAP results	Search Base and Role Search Base carried literal double quotes from 8.1. 8.3 compares them against returned DNs.	Remove the quotes, either in 8.1 before upgrading or in 8.3 after migration
Roles not fetched. User Role Name Attribute shows sAMAccountName instead of cn, and changing it also changes User Name Attribute	Gateway Web UI edit panel	The panel maps both attributes to the same field	Fixed starting with Beta4. Before that, edit through the OpenAPI endpoint.
Some or all memberOf roles silently missing	Gateway role filtering	The role DN falls outside the configured role search base	Correct or widen roleSearchBase
Could not read SSL key file /etc/step/certs/ssl-cert.key.pk8".	Gateway TLS client, loading the client private key	A trailing " became part of the sslkey path	Remove all quotes and escapes from the connection properties
The server requested password-based authentication, but no password was provided.	PostgreSQL authentication stage, after TLS	With the password empty, 8.3 does not send the username. The driver connects as the OS account that runs Ignition.	Set the embedded password to a single space. Still present in 8.3.0-rc1; the fix is expected in 8.3.1.

Where does the Postgres connection request stop: file read, TLS, or authentication?

The full stack trace traces the request hop by hop.

1. The faulted-datasource retry task (DatasourceManagerImpl$FaultedDatasourceRetryer) calls DatasourceImpl.runTest.
2. That call goes into Apache DBCP2 BasicDataSource.createPoolableConnectionFactory.
3. DBCP2 calls the PostgreSQL JDBC driver: Driver.connect, then ConnectionFactoryImpl.enableSSL, then MakeSSL.convert.
4. The driver hands off to the JVM's TLS stack.

Two frames in the TLS stack locate the fault:

• Finished$T13FinishedConsumer.onConsumeFinished shows the client had already processed the server's TLS 1.3 Finished message.
• CertificateMessage$T13CertificateProducer shows it was building its own Certificate message in reply to the server's certificate request.

That sequence rules out the whole network side:

• TCP to port 5432 connected.
• The server certificate chain validated against sslrootcert.
• The verify-full hostname check passed.
• The server asked for a client certificate.

The failure is local to the gateway: org.postgresql.ssl.LazyKeyManager.getPrivateKey could not load the key. No client credential ever went on the wire.

Innermost Caused by	Meaning	Next check
PSQLException: Could not read SSL key file ... from LazyKeyManager	TLS reached client-certificate selection, then the key load failed	File readability, then the literal path string
The server requested password-based authentication, but no password was provided.	TLS completed; the server's pg_hba.conf rule for the presented role demands a password	Which role the driver actually presented
Connection refused or timeout before any TLS frames	Network or listener problem	Routing, firewall, and the listen_addresses setting on the server

Rule out the filesystem first. The service account on the upgraded node is the one that matters, and a reinstall is the point where ownership and run-as user can drift.

1. Find the account the gateway JVM runs as: ps -eo user,args | grep -i [i]gnition.
2. Confirm the file exists with the expected mode: ls -l /etc/step/certs/.
3. Test read access as that account: sudo -u <gateway_user> test -r /etc/step/certs/ssl-cert.key.pk8 && echo readable.
4. If the test fails, correct ownership or group read permission and retry the connection.
5. If the test passes, move to the path-string check.

In this installation the file was present. The 8.1 backup node kept reading the same path and holding a valid connection. That points to the upgraded node's stored configuration, not to the certificate material or the CA chain.

Does the key path handed to PgJDBC end in a quote character?

Read the error text literally: .../ssl-cert.key.pk8".. The final period ends the driver's message. The double quote before it is part of the path. The underlying exception confirms this: java.io.FileNotFoundException: /etc/step/certs/ssl-cert.key.pk8". The driver was trying to open a file whose name ends in a quote.

Pull the stored configuration from the gateway config API (the database-connection resource) and look at connectionProps. The faulted connection returned:

"connectionProps": "\"?ssl=true;sslmode=verify-full;sslrootcert=/etc/step/certs/root_ca.crt;sslcert=/etc/step/certs/ssl-cert.pem;sslkey=/etc/step/certs/ssl-cert.key.pk8\""

The escaped quotes are stored characters. The extra-properties field is split on ;, and each key=value pair goes to the driver as-is. There is no shell-style quote stripping and no backslash unescaping. So:

• The leading " becomes part of the first key, turning it into "?ssl.
• The trailing " becomes part of the last value, the sslkey path.
• A space after the filename just adds a space to the path, next to the quote.
• Single quotes, double quotes, and backslash escaping all reach the driver as literal characters.

In this installation the quotes were not in the imported configuration. They were added by hand after the upgrade, because the unquoted string failed with a password-authentication error. The quotes did not fix anything. They moved the failure earlier: a broken key path aborts the handshake at client-key load, before the server ever evaluates authentication. The error changed, but the connection got no closer to working.

Decision:

• If any quote or escape character appears in connectionProps, remove all of them and save.
• If the fault then becomes the password-authentication error, go to the role check.

Which database role does the gateway present after the handshake?

With clean properties, the connection completes TLS and then fails with The server requested password-based authentication, but no password was provided.. Here is how that happens:

1. Between 8.1 and 8.3, the gateway changed how it builds the connection request. When a username is configured and the password field is empty, 8.3 does not send the username.
2. With no user property, the PostgreSQL JDBC driver falls back to the JVM's OS user name, which is the account running Ignition.
3. The server looks up pg_hba.conf for that OS-account role, not the configured database role.
4. The cert rule written for the intended role does not match. The connection falls through to a password-based rule.
5. The driver has no password to send, so the connection fails.

Even if a cert rule did match, it would fail: certificate authentication requires the client certificate's CN to match the requested role, directly or through a user-name map.

Setting	8.1 behavior	8.3 behavior (through 8.3.0-rc1)
Username set, password empty	Username sent; cert auth runs as the configured role	Username not sent; the driver presents the gateway OS account
Username set, password non-empty	Username and password sent	Username and password sent; cert auth runs as the configured role

Confirm this branch on the server side. Enable log_connections on the PostgreSQL server, retry the connection, and read the role name logged for the failed attempt. If the log shows the gateway's Linux service account instead of the configured username, this is the fault. Fixing file paths or TLS parameters will not change the result.

A non-empty password makes 8.3 send the username again. Setting the embedded password to a single space is enough. Under a cert rule the server never issues a password challenge, so the space is never transmitted or checked. Authentication rests on the certificate and the role match. This workaround produced a valid connection on the affected gateway. The underlying fix did not ship in 8.3.0-rc1 and is expected in 8.3.1.

Are the remaining connection properties valid for PgJDBC?

The confirmed working string on 8.3 was:

?ssl=true;sslmode=verify-full;sslrootcert=/etc/step/certs/root_ca.crt;sslcert=/etc/step/certs/ssl-cert.pem;sslkey=/etc/step/certs/ssl-cert.key.pk8

Property	Function on the path	Failure it produces if wrong
?ssl=true	Requests TLS. The ? is URL query syntax, and in this field it becomes part of the key name.	Harmless here, because sslmode other than disable already enables TLS in PgJDBC
sslmode=verify-full	Validates the server chain and matches the host in the connect URL against the server certificate CN/SAN	Hostname-mismatch or chain errors before the client certificate stage
sslrootcert	CA used to verify the server certificate	Path-validation failure
sslcert	Client certificate sent when the server asks for one	Server rejects the client certificate, or it fails the CN-to-role match
sslkey	Client private key read by LazyKeyManager	Could not read SSL key file ...

The leading ? was left in the working string and caused no fault. Dropping it gives a cleaner key name, since the TLS decision comes from sslmode.

The .pk8This key already worked under 8.1, so the format was not the problem here. When rotating certificates, regenerate the key file with:

openssl pkcs8 -topk8 -inform PEM -outform DER -in ssl-cert.key -out ssl-cert.key.pk8 -nocrypt

After regenerating, reapply read access for the gateway account.

Do the AD search bases still parse as distinguished names?

The 8.1 user source had working values in Search Base and Role Search Base, but each value was wrapped in double quotes. The migration carried the quotes into 8.3 unchanged. In 8.1, the quoted base entry did not visibly affect the JNDI search.

8.3 added nested group membership support, and it uses the search bases in two ways:

1. As the base entry for the JNDI search. This behavior existed in 8.1.
2. As a containment filter. Every user DN and group DN returned during membership lookup must fall under the search base for its entity type (user or role). Entries outside the base are ignored.

The containment check is a DN comparison. DNs returned by the LDAP server never carry quote characters, so a quoted base matches nothing. Every user and group gets discarded, and the profile resolves nothing, even though the directory answered correctly.

Read the stored values directly, not from the rendered form. Any of these three sources works:

• The config API user-source resource, visible through the gateway OpenAPI documentation.
• The JSON payload the Web UI loads when you open the edit panel, viewed in browser developer tools.
• The resource file on disk: home/data/config/resources/core/ignition/user-source/<PROFILE_NAME>/config.json.

Decision:

• If either base contains quote characters, remove them.
• If the bases are clean and roles are still missing, check role scope next.

For planned upgrades, strip the quotes in 8.1 before taking the gateway backup. The migrated 8.3 configuration then starts clean.

Do the returned role DNs fall inside the Role Search Base?

The same containment filter drops real groups when the role search base is narrower than the directory layout. For example:

Field	Value	8.3 result
roleSearchBase	ou=orgunit,dc=acme,dc=org	-
User's memberOf	cn=role1,dc=acme,dc=org	Role ignored; the DN is not under ou=orgunit
User's memberOf	cn=role2,ou=orgunit,dc=acme,dc=org	Role returned

In 8.1 the first group might have resolved. In 8.3 it disappears without an error.

To see whether the directory returns the values at all, temporarily map the raw attribute into a field that bypasses role filtering. Setting emailAttribute to memberOf exposed the full group list on one affected profile. The same attribute in groupRoleAttribute produced an empty role list.

That split means the LDAP query and the bind are fine, and the drop happens in role resolution. Compare each returned group DN against roleSearchBase character by character. Look for extra RDN components, stray whitespace, and quote characters. Restore emailAttribute after the test.

Did the AD edit panel overwrite the User Name Attribute?

Before Beta4, the 8.3 Web UI edit panel for AD user sources mapped User Name Attribute and User Role Name Attribute to the same underlying setting. Setting the role name attribute back to cn silently changed the user name attribute. The profile then showed sAMAccountName where cn was expected, and roles would not resolve.

The migration was not the cause. A restored 8.1 backup, re-migrated and read directly, showed correct values: "groupRoleAttribute": "memberOf" and "roleNameAttribute": "cn". The corruption comes from saving through the panel.

The panel bug is fixed starting with Beta4. On builds before that, edit the resource through the API instead of the panel:

1. Log in to the gateway and open http://<YOUR_HOST>:<YOUR_PORT>/openapi. The documentation requires an authenticated session.
2. Search for the modify user sources operation, or expand the config user-source section.
3. Get a correctly shaped payload by saving any harmless change in the Web UI and copying the request body from the browser developer tools.
4. In the copied payload, set config.settings.userNameAttribute and config.settings.roleNameAttribute to their intended values.
5. Send the request. An external HTTP client needs a gateway API token. A browser-based client can reuse the existing session cookies.
6. Re-read the resource through the API or from config.json on disk, and confirm both attributes hold distinct, correct values.

How do you apply the fixes and prove both paths work?

Upgrade the primary node first, as this installation did. The 8.1 backup keeps serving authentication and database traffic, and gives you a known-good reference for every comparison.

1. On 8.1, open the AD user source and remove any quote characters from Search Base and Role Search Base. Confirm AD logins still work on 8.1, then take a fresh gateway backup.
2. Upgrade the primary. Before opening any edit panels, read the migrated AD resource from config.json or the config API. Check the two search bases, userNameAttribute, roleNameAttribute and groupRoleAttribute.
3. If the build is earlier than Beta4, make AD attribute changes through the OpenAPI modify operation only.
4. Open the PostgreSQL connection. Strip every quote and backslash from the extended connection properties so the field matches the working string exactly.
5. Keep the configured username. Set the embedded password to a single space and save. Leave it in place until you run a build that sends the username with an empty password; per the fix schedule, 8.3.0-rc1 does not, and 8.3.1 is the target.
6. As the gateway service account, confirm read access to root_ca.crt, ssl-cert.pem and ssl-cert.key.pk8.
7. Check that the database connection status on the gateway reads Valid, not Faulted. Check the PostgreSQL server log: the connection should be authenticated for the configured role, not the Linux service account.
8. From a gateway-scoped query (for example the Designer database query browser), run SELECT current_user, ssl, client_dn FROM pg_stat_ssl WHERE pid = pg_backend_pid();. Expect three things: current_user equals the configured username, ssl is true, and client_dn carries the subject of ssl-cert.pem.
9. Authenticate a known AD account against the user source on the upgraded primary. Confirm the returned role list matches the list the same account gets on the 8.1 backup, including groups whose DNs sit deepest in the role search base.

FAQ