Very excited for the addition of Kafka to Ignition, wanted to ask if anyone’s found anything regarding how Ignition’s implementation of Kafka will support network outages. Specifically, Kafka producers have in memory buffers for storing messages until they’re successfully emitted, but haven’t seen any docs around whether this is the case in Ignition. Also unsure how or if Store & Forward plays into as well or is duplicative in function.
The Kafka producers in Ignition will definitely batch records using it’s internal memory buffer as you mentioned. We don’t use Store and Forward in Event Streams yet, but we do plan on incorporating it in the future.
1 Like
On gateway restart, are the unsent events stored somewhere and then restored?
Appreciate the info! Just curious but is there any information about how large the buffer will be/is that something that can be configured? Just want to understand limits to estimate how long of an outage our system would handle
No, unfortunately they’re gone. We do plan on adding some sort of persistence eventually, probably through S&F.
The default memory buffer for Kafka producers is 32MB, but this can be adjusted through the Kafka Connector settings producer props. buffer.memory is the property you’d have to adjust.
1 Like