Connecting to Kafka broker

Ignition
1

I'm just playing with kafka event streaming. Following the "Kafka - source example" in the IU library I've tried to connect to a kafka server, but the Kafka connector appears as faulted.
Next my checks:

  • Kafka and Ignition 8.3 are both running as docker containers

  • Both are connected to the same docker bridge network

  • If I execute a bash in each container I can ping the other container

  • If I run netstat in the Kafka container, connection to port 9092 is established and the foreign address is the Ignition container

  • If I run ss -te in the Ignition container I also see that there is an established connection to <kafka_ip>:9092

  • If I run nc <kafka_ip>:9092 in the Ignition container, I can establish a connection

  • My kafka connection in Ignition is configured to connect to
    image
    where kafka-broker is the name of the kafka container - I tried also with the kafka container IP on the docker network, but with no success

  • If I run tshark in the Kafka container, telegrams from/to Ignition are just

    image
    image684×142 26.2 KB

  • If I use KafkIO which can connect to Kafka server, requests to the server go further then that

All above said, my result is

image

Any suggestion on what to check further?

Thanks in advance, regards

2

For anyone intersted in this topic, I managed to connect to the dockerized Kafka broker: the problem was the listeners configuration. The right docker run command in my configuration is:

docker run \
-p 9092:9092 \
-e KAFKA_PROCESS_ROLES="broker,controller" \
-e KAFKA_NODE_ID=1 \
-e KAFKA_CONTROLLER_QUORUM_VOTERS="1@localhost:29093" \
-e KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER" \
-e KAFKA_LISTENERS="INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093" \
-e KAFKA_ADVERTISED_LISTENERS="INTERNAL://broker:29092,EXTERNAL://localhost:9092" \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" \
-e KAFKA_INTER_BROKER_LISTENER_NAME="INTERNAL" \
--network net-ign \
apache/kafka:latest

By running the Kafka container that way, a dockerized Ignition which is attached to the same docker bridge network as Kafka (in my casenet-ign) can connect to bootstrap-server broker:29092, while an external client like KafkIO can connect to localhost:9092, where 9092is the published port.