Event Streams to Kafka: the file-based config shapes, and five silent failure modes to know about

I have been running 8.3.8 Event Streams publishing to a Kafka-compatible cluster (Redpanda), authoring the streams as project files rather than in the Designer, and hit a few things worth sharing since I could not find them documented anywhere.

The Kafka handler's topic / key / value fields are ExpressionValue objects ({"data": ..., "type": "STATIC" | "EXPRESSION"}) - a bare string parses without complaint and produces nothing. Tag identity lives in event metadata ({event.metadata.tagPath}), not in data. sourceEncoder, batch, filter, transform and onError are all required, and each missing one is a separate NPE at stream creation. Batch timers set to 0 are disabled, not fast - the stage then flushes only at maxQueueSize, which at low tag rates looks exactly like a dead pipe. And a file-based config change needs Scan File System twice: the first scan shuts the stream down, the second one starts the replacement.

The silent one to watch: failureStrategy: IGNORE reports the handler GOOD with zero errors while delivering nothing, and a malformed handler entry unregisters the whole stream with no log line at all.

Happy to share the working config.json shapes if useful. Full write-up with the measurements (including a kill battery: broker down, database down, gateway restart, each loss-accounted against a 1 Hz heartbeat tag) is here: Event Streams to TimescaleDB: the config recipe, five silent failures, and a loss-accounted kill battery — Aradh Suresh

Some of this may be specific to Event Streams 1.3.8 - corrections welcome if anyone from IA sees this.

1 Like