apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Support new major kafka release 0.9.0
Date Mon, 16 Nov 2015 18:53:31 GMT
I will be working on rewriting the kafka input operator.

Here is the ticket
https://malhar.atlassian.net/browse/MLHR-1904

Here is some comments on the ticket

The RC2 is out here
https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/

We will keep most features of the old input operator but the internal
mechanism will be changed, for example, using new API to refresh the
metadata
The bugs that will be fixed:

   - Synchronized offset checkpoint
   - Transient offsetmanager

New features:

   - Support customized partition schema
   - Default OffsetManager using new

Improvement

   - Add window id and application name to OffsetManager interface
   - Support multi-topic
   - Easy configuration


Please leave thoughts here or on the ticket. Thanks

Best,
Siyuan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message