Configuring Ignition Kafka Connections for Stream Pipelines

Gateway Build and Module Licensing

The Designer can offer a Kafka source for a stream pipeline before the gateway can hold a Kafka connection. The Designer lists the source type as soon as the Kafka module is installed. The gateway-side connection type is a separate requirement. On pre-release builds earlier than 20250528, an installed Kafka module showed the Kafka source in the Designer, but no Kafka Service Connector appeared on the gateway. From build 20250528 onward, the connector type is present. If you have a Designer source with no gateway connector, upgrade the gateway build. Changing configuration will not fix it.

Confirm licensing before you commission. The module list now includes several new entries that are billed differently:

An unlicensed paid module runs in trial mode. When the trial expires, the module stops. On a Kafka pipeline this looks like consumption that stops for no clear reason. Activate the license before you start any soak test.

Check 1: On the gateway modules page, expect the Kafka module to show as running. It should be activated, or in trial with the trial state known.
Check 2: Expect the gateway build to be 20250528 or later, and Kafka to be offered as a connection type in the next section.

Kafka Service Connector Connection

The term Service Connector here means a gateway-scoped connection object that holds broker addresses, security settings and credentials. Stream pipelines do not carry broker details. They reference the gateway connection by name, so secrets stay on the gateway and are not stored in project resources.

1. Open the gateway web interface and go to Connections > Service Connectors > Connections.
2. Create a new connection and select the Kafka type.
3. Give it a stable name. Pipelines bind by name, so renaming it later means re-pointing every pipeline source that uses it.
4. Enter the bootstrap broker list as host:port pairs. Enter at least two brokers on a multi-broker cluster so the bootstrap still works if one broker is down.
5. Set the security options (next two sections) and save.

Check 3: Expect the new connection in the Service Connectors list with a status indicator. A faulted status at this stage is normal until the network path and security are correct.

Broker Addressing and Network Path

A Kafka client connects in two stages:

1. It connects to any bootstrap broker and requests cluster metadata.
2. It connects directly to each partition leader, using the address that broker advertises in that metadata.

The gateway therefore needs to reach every broker's advertised host and port, not only the bootstrap address. The most common failure is a broker that advertises localhost or a container-internal hostname. The bootstrap succeeds, and every fetch after it times out.

Run these tests from the gateway host, not from an engineering laptop. Replace the hostnames and port with your broker listener values. Port 9092 is only the conventional plaintext default.

# PowerShell on the gateway host
Resolve-DnsName broker1.plant.local
Test-NetConnection -ComputerName broker1.plant.local -Port 9092
Test-NetConnection -ComputerName broker2.plant.local -Port 9092

Check 4: Expect every advertised broker hostname to resolve to a routable address from the gateway host.
Check 5: Expect TcpTestSucceeded : True for every broker. A single failure here explains partial consumption later.

Security Protocol and Credentials

The security protocol in the connection must match the listener it points to. The options are plaintext, TLS, or SASL over TLS. The failure modes are distinct:

• TLS client on a plaintext listener, or the reverse: the connection drops at handshake.
• SASL mechanism mismatch: the broker rejects authentication. For example, the client uses PLAIN while the broker accepts only SCRAM.
• Untrusted certificate chain: the gateway logs an SSL handshake error.

For TLS, the gateway must trust the broker certificate chain. The certificate subject or SAN must also match the advertised hostname the client actually connects to. Matching only the bootstrap name is not enough.

Authorization is separate from authentication. A client can log in successfully and still fail when it requests metadata or offsets. For a consuming pipeline, the principal needs these permissions:

• Describe and Read on the topic
• Read on the consumer group
• Write on the topic, only if the pipeline also produces

Check 6: Expect a healthy status on the Service Connector connection. Expect the gateway logs to show no handshake, authentication or authorization errors from the Kafka client after the connection stabilizes.

Stream Pipeline Source Binding

In the Designer, open the stream pipeline and set its source to Kafka. Then configure it:

1. Select the gateway connection by the name you gave it.
2. Set the topic.
3. Set the consumer group and start position, where the source exposes those settings.

Consumer group behavior decides what the pipeline actually receives:

• Kafka stores committed offsets per group.
• Two gateways, or a production and a test gateway, that share a group ID split the partitions between them. Each sees only part of the stream.
• A new group that starts at the latest offset receives nothing until new messages arrive, which is easy to mistake for a dead connection.

Give each pipeline instance its own group ID unless you deliberately want load sharing.

Check 7: Expect the gateway connection to appear in the source's connection selector. If the selector is empty while the Kafka source type is available, the gateway connection is missing or the gateway build predates 20250528. Go back to the first two sections.

End-to-End Verification

These commands use the standard Kafka CLI tools. The hostname, topic and group names are placeholders.

kafka-console-producer.sh --bootstrap-server broker1.plant.local:9092 --topic ignition-test
kafka-console-consumer.sh --bootstrap-server broker1.plant.local:9092 --topic ignition-test --from-beginning --group verify-cli
kafka-consumer-groups.sh --bootstrap-server broker1.plant.local:9092 --describe --group <pipeline-group>

1. Produce: send a test record to the topic. Expect the producer to accept input without errors.
2. Broker confirmation: read the record with the console consumer under a separate group. Expect the record to print. This proves the topic and broker work independently of the gateway.
3. Pipeline receipt: check the pipeline's status or its downstream handler. Expect the record to appear once for each record produced.
4. Group state: run kafka-consumer-groups.sh --describe on the pipeline's group. Expect assigned partitions with a consumer ID from the gateway client, and LAG returning to 0 after the pipeline catches up.
5. Restart behavior: restart the pipeline, then produce one more record. Expect consumption to resume from the committed offset without replaying the whole topic. Under at-least-once delivery, a small replay window is normal, so downstream handling must tolerate duplicates.

FAQ