flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetzger <...@git.apache.org>
Subject [GitHub] flink pull request: Cleanup of low level Kafka consumer (Persisten...
Date Tue, 17 Mar 2015 22:17:30 GMT
Github user rmetzger commented on the pull request:

    First of all: Great work @gaborhermann! The whole Kafka connector code is well written
and contains logic for a lot of cases. I think it was already quite some work to get it to
the state it currently is.
    I took this pull request as a base to add an integration test which is starting a local
Zookeeper and Kafka instance.
    I also fixed a critical issue in the streaming code, regarding exception forwarding.
    My current WIP is here: https://github.com/rmetzger/flink/tree/kafka-gabor
    The code needs some more care until its in a good state. The following is missing:
    - More integration tests: Testing of the `PersistentKafkaSource` with taskmanager failure
and job restart, also tests for other stuff (partitioner as class & as instance)
    - Add means to `KafkaSource` and `KafkaSink` for users to pass their own configuration
properties to Kafka.
    - There are some TODOs in the code
    - Testing of the code in a distributed setup.
    I'll work on the cluster testing tomorrow. In the course of that, I'm probably going to
add more code to my "wip" branch.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message