On an Ignition Maker Edition 8.3.0-beta2 gateway, Perspective logins through OIDC against a Microsoft Entra tenant stall for a few seconds on the first three or four logins of the day, then complete quickly. The same project on 8.1 (approximately 8.1.44) logged in almost instantly for months. Gateway tracing showed one slow login's back-channel token exchange took 5 seconds. A later warm login returned the bearer token in 108 ms. The delay is between the gateway and Microsoft, not in Perspective rendering. The job is to find which leg of that outbound path goes cold.
Which hops does an Ignition OIDC login actually cross?
An OIDC authorization-code login has two separate network paths. The front channel runs through the operator's browser. The back channel runs directly from the gateway to the identity provider. The two paths use different DNS resolvers, different routes, and different TLS stacks. A delay on one path is invisible to tools that watch only the other.
| Hop | Sender → Receiver | Channel | Where its time shows up |
|---|---|---|---|
| 1 | Browser → Ignition gateway (Perspective session, login trigger) | Front | Browser performance recording |
| 2 | Gateway → browser (redirect to Entra authorize endpoint) | Front | Browser performance recording |
| 3 | Browser → Entra (sign-in, SSO cookie, MFA if enforced) | Front | Browser performance recording |
| 4 | Entra → browser → gateway redirect URI carrying the authorization code | Front | Browser performance recording (callback request) |
| 5 | Gateway → Entra token endpoint (code exchanged for tokens) | Back | Gateway log: Received HTTP Response in %s
|
| 6 | Gateway → Entra user info endpoint (only if configured) | Back | Included in the same log message |
| 7 | Gateway → browser (authenticated session, Perspective loads) | Front | Browser performance recording |
Hops 5 and 6 run while the browser's callback request from hop 4 is still open. A slow back channel therefore looks like a long pending request on the redirect URI, even though the browser does not cause the delay. The gateway also uses the same outbound DNS and TLS path for any signing-key retrieval needed to validate the ID token.
Check: Open the gateway's OIDC identity provider configuration. Record the authorization, token, and user info endpoint hostnames. You will test each back-channel host by name later.
Is the 8.1 versus 8.3 comparison actually like-for-like?
The Ignition configuration matches because the 8.3 gateway was restored from an 8.1 gateway backup. The platform does not match. 8.1 ran on a Raspberry Pi 4, and 8.3 runs on a Raspberry Pi 5. The two units form a cold-standby pair on the same IP address and never run at the same time. The "8.1 was fast" baseline therefore changes several variables at once.
| Variable | 8.1 baseline | 8.3 current | Can it add back-channel latency? |
|---|---|---|---|
| Ignition version / HTTP client | 8.1 (approx. .44) | 8.3.0-beta2 |
Yes: connection pooling, keep-alive, TLS setup |
| Hardware | Raspberry Pi 4 | Raspberry Pi 5 | Indirectly: NIC, link negotiation, Wi-Fi vs cable |
| OS image / resolver config | Pi 4 image | Pi 5 image | Yes: DNS servers, IPv6 preference, resolver timeout |
| Bundled Java runtime | Shipped with 8.1 | Shipped with 8.3 | Yes: JVM DNS caching and TLS defaults |
| Gateway config, Entra tenant, endpoints | Identical (backup/restore) | Ruled out as a difference | |
Check: Record which of these variables you can change one at a time. The cold-standby Pi 4 is still available, which makes a split test possible later.
Is the physical link on the Pi 5 clean before blaming the protocol?
A few-second stall on the first request after an idle period is a classic sign of a link that sleeps or renegotiates. Rule this out before capturing traces.
- Run
ip -s linkand look for non-zero RX/TX errors or drops on the active interface. - On wired Ethernet, run
ethtool eth0and confirm the negotiated speed and full duplex match the switch port. - On Wi-Fi, run
iw dev wlan0 get power_save. Wi-Fi power saving delays the first packets after idle. Put the gateway on a wired link, or disable power save and re-test. - Compare the Pi 5 IP settings (gateway, DNS servers) with the Pi 4. The two units share an IP, but DHCP or static config may still differ.
Check: The interface shows no errors, and the link type and settings match the Pi 4 or are documented as different.
How do you get the gateway to report back-channel timing?
The logger gateway.HttpOIDCClientService reports how long the back channel took. It emits a message in the form Received HTTP Response in %s. The value covers the time from sending the back-channel request(s) to receiving the response(s). That includes the authorization-code-for-token exchange and the optional user info call.
- In the gateway's logging configuration, set
gateway.HttpOIDCClientServicetoTRACE. - Do not restart the gateway. A restart resets the cold state you are trying to capture.
- Leave tracing on overnight, so the first logins of the next day are recorded.
- After each test login, copy the wrapper log from the gateway's logs directory before it rotates.
Check: Log in once. Search the wrapper log for Received HTTP Response in, and note the timestamp and duration. A warm login on this installation measured 108 ms. That is the reference for "healthy".
How do you capture the browser side without masking the cold path?
The delay appeared on the first three or four logins and then vanished once recording started. Capture timing matters as much as capture method.
- Before the first login of the day, open a fresh browser session (private window or clean profile) with no cached Entra session.
- Open the browser dev tools and start a performance recording, including network timing.
- Navigate to the Perspective project and log in through the OIDC identity provider.
- Once the browser returns to Ignition in a logged-in state, stop the recording and save it.
- Save the wrapper log covering the same minute.
- Repeat the capture on the 8.1 gateway (Pi 4) under the same conditions, so you have a matched pair.
Check: The recording shows the full sequence: Ignition page, redirect to Entra, Entra sign-in, callback to the gateway redirect URI, Perspective load. Find the request with the largest wall-clock time.
Where does the delay land: front channel or back channel?
Line up the browser recording and the gateway TRACE line by timestamp, then read the result from this table.
| Observation | Leg | Probable cause | Next check |
|---|---|---|---|
Callback request pending for seconds; Received HTTP Response in shows seconds (5 s here) |
Back channel (hops 5–6) | Gateway outbound DNS, TCP connect, TLS handshake, or Entra token endpoint latency | Shell timing tests from the Pi (next section) |
| Trace and callback are fast; long gap before Perspective renders | Gateway (hop 7) | Session or project startup on the new hardware | Gateway performance and status pages |
| Slow only on the first logins of the day, fast afterwards | Whichever leg is slow | Cold state: expired DNS cache, closed pooled connections, idle link | Repeat timing tests after an idle period |
On this installation, the evidence points to row 1 combined with row 4: a back-channel exchange that is slow when cold and fast when warm.
Check: Confirm that a slow login's callback request duration roughly matches the traced back-channel time. If it does, the stall is entirely on the gateway-to-Entra path.
Why would the token exchange take 5 seconds cold and 108 ms warm?
A cold back-channel request pays for every setup step: DNS resolution, TCP connect, TLS handshake, then the HTTP exchange. Once connections and cache entries are warm, only the HTTP exchange remains, which fits the 108 ms figure. Break the 5 seconds into those parts directly from the Pi. Use curl, because it runs on the same OS resolver and route but bypasses Ignition.
| Dominant timer | Meaning | Corrective action |
|---|---|---|
time_namelookup |
Slow DNS resolution | Check nameservers and the options timeout: / attempts: values in /etc/resolv.conf. A round multi-second stall usually means the first nameserver timed out and a second answered. Remove dead resolvers, and fix broken IPv6 (AAAA) resolution or routing. |
time_connect minus lookup |
Slow TCP connect | Check firewall, proxy, or routing between the Pi 5 and the internet. Compare -4 and -6 results. |
time_appconnect minus connect |
Slow TLS handshake | Check certificate revocation lookups and TLS inspection devices in the path. |
time_starttransfer minus appconnect |
Microsoft server response time | Outside the plant. Correlate with Entra sign-in log timestamps. |
A warm curl result tells you little. Run the test first thing in the morning, or after the Pi has been idle as long as it is before the first login. The Java runtime keeps its own DNS cache, separate from the OS. If curl stays fast while the gateway trace stays slow, compare the JVM DNS caching settings in the Java security properties of the runtime bundled with each Ignition version.
Check: A cold-state curl run reproduces a multi-second total, and one timer accounts for most of it. That timer names the layer to fix.
How do you split Ignition version from Pi hardware on a cold-standby pair?
Only one unit can hold the production IP at a time. Split the test so that each run changes only one variable.
- Boot the Pi 4 on a temporary IP. Run the same cold-state curl test against the token endpoint host at the same time of day as the Pi 5 test.
- Compare the results. Slow on the Pi 5 and fast on the Pi 4 means the cause is the OS image, resolver, or link. Fix it at the OS level; Ignition is not involved.
- If curl is fast on both units but the Pi 5 gateway trace is still slow when cold, the gateway's HTTP client or JVM is responsible. Swap the Pi 4 back onto the production IP running 8.1. Capture several first-of-day logins with the same trace and recording procedure.
- If 8.1 stays fast on the same network where 8.3 is slow, send both wrapper logs and both browser recordings to Inductive Automation support.
8.3.0-beta2is pre-release software. Re-run the same capture after upgrading to a later 8.3 build before changing anything else.
Check: You have one matched pair of captures in which a single variable changed and the delay moved with it.
How do you prove the cold-start delay is gone?
- Apply the fix at the layer that step 7 identified: resolver, IP family, link power management, or a gateway upgrade.
- Leave
gateway.HttpOIDCClientServiceatTRACEand let the gateway sit idle overnight. - For the first four logins of the next day, record the
Received HTTP Response invalue. Each should be close to the warm figure (108 ms on this installation), not several seconds. - In a matching browser recording, confirm that the callback request to the gateway redirect URI no longer stays pending for seconds.
- Repeat for at least several consecutive days, because the original fault appeared daily and only on the first logins.
- Set
gateway.HttpOIDCClientServiceback to its default level. Log in once more from a fresh browser session and confirm that Perspective opens without a visible pause at the authenticating stage.
Frequently asked questions
Does a slow Ignition OIDC login always mean Microsoft Entra is slow?
No. The gateway's back-channel time includes its own DNS lookup, TCP connect, and TLS handshake to Entra. Time those steps from the gateway host with curl; only a large time-to-first-byte after the TLS handshake points at the Microsoft server.
Can I see the OIDC token exchange time in the Ignition gateway logs?
Yes. Set logger gateway.HttpOIDCClientService to TRACE and look for Received HTTP Response in %s. The value covers the code-for-token exchange plus the optional user info request.
Can I compare Ignition 8.1 and 8.3 OIDC performance on different hardware?
Only after separating the variables. Run the same cold-state curl test on both hosts to isolate OS and network differences first. Then compare gateway traces with only the Ignition version changed.
Does the browser dev tools recording show the gateway-to-Entra token request?
Not directly. The token exchange is server-to-server, so it appears in the browser only as a long pending callback request to the gateway redirect URI. Use the gateway trace to get the back-channel duration itself.
Why is the OIDC login slow only on the first logins of the day?
After an idle period, the gateway has to rebuild its cached DNS entries and pooled HTTPS connections, and a sleeping network link has to wake up. Each of these costs time on the first request. Test from the gateway host after the same idle period to find which setup step accounts for the delay.