Will it be possible to seek by timestamp?
I’m thinking about potential for replay of historical events.
We don’t have support for that yet. However, given enough interest, we could probably add something like that. How would you be consuming these historical events?
1 Like
How would you be consuming these historical events?
Haven’t worked that part out yet, don’t have a specific use case but trying to explore the possibilities. In terms of what would be returned from a new system function, would essentially be a dataset similar to that returned from system.kafka.seekLatest. I’m no Kafka expert yet, but something tells me there is enormous potential in the Kafka / Event Streams space.
Basically, thinking some cross between system.kafka.pollPartition and system.kafka.seekLatest, but querying for the nearest offset(s) by timestamp. Would pass in either a pair of timestamps, or a start time plus number of records. I believe there is something in the Kafka API for identifying an offset by timestamp, offsetsForTimes()? With an additional system function behaving like seekLatest, could probably do plenty of cool things via scripting, bindings and expressions, leaving implementation to users. If there were a separate function that just returned the offset associated with a timestamp, there mightn’t be a huge amount of difference required from system.kafka.seekLatest, to accept a starting offset - could leave it to the user to evaluate the start and count of records by querying offsets for themselves.
But perhaps there is some end state that could function in a similar manner to tag history based playback demonstrated https://youtu.be/GdNr47eY1IM . Don’t know how it would scale and loads of other questions around performance and retention windows, but if topics were structured accordingly could potentially have Kafka as a tag source and (at least short term) history?