kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Labbe <mrla...@gmail.com>
Subject Re: Consumer re-design and Python
Date Sat, 16 Feb 2013 01:29:20 GMT
Thanks for the info David.

That was my understanding was well for what you pointed. Even though I
didn't mention it, I has also seen your kafka-python but I felt it was a
little too low level for my needs. Do you have a branch for 0.8 support on
github (didn't see any)?


On Mon, Feb 11, 2013 at 10:31 AM, David Arthur <mumrah@gmail.com> wrote:

> On 1/31/13 3:30 PM, Marc Labbe wrote:
>> Hi,
>> I am fairly new to Kafka and Scala, I am trying to see through the
>> consumer
>> re-design changes, proposed and implemented for 0.8 and after, which will
>> affect other languages implementations. There are documentation pages on
>> the wiki, JIRA issues but I still can't figure out what's already there
>> for
>> 0.8, what will be there in the future and how it affects the consumers
>> written in other languages (Python in my case).
>> For instance, I am looking at
>> https://cwiki.apache.org/**KAFKA/consumer-client-re-**design.html<https://cwiki.apache.org/KAFKA/consumer-client-re-design.html>and
the very
>> well documented
>> https://cwiki.apache.org/**confluence/display/KAFKA/**
>> Kafka+Detailed+Consumer+**Coordinator+Design<https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design>
>> and
>> I am not sure what part is in the works, done and still a proposal. I feel
>> there are changes there already in 0.8 but not completely, referring
>> especially to KAFKA-364 and KAFKA-264.
> https://cwiki.apache.org/**confluence/display/KAFKA/**
> Consumer+Client+Re-Design<https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design>
> and
> https://cwiki.apache.org/**confluence/display/KAFKA/**
> Kafka+Detailed+Consumer+**Coordinator+Design<https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design>
> are the current design docs (as far as I know).
>> Is this all accurate and up to date? There are talks of a coordinator as
>> well but from what I see, this hasn't been implemented so far.
> From my understanding, the client redesign has not been finalized and it
> still in-progress/todo.
>> After all, maybe my question is: other than the wire protocol changes,
>> what
>> changes should I expect to do to SimpleConsumer client written in Python
>> for v0.8? What should I do next to implement a high level consumer
>> (ZookeeperConsumerConnector?) which fits with the design proposal?
> With 0.8, you will not need to connect to ZooKeeper from the clients. With
> KAFKA-657, offsets are centrally managed by the broker. Any broker can
> handle these requests.
>> Has anyone started making changes to their implementation yet (thinking
>> Brod or Samsa)? I'll post that question on github too.
> I am working updating my Python client: https://github.com/mumrah/**
> kafka-python <https://github.com/mumrah/kafka-python>, still a ways to go
> yet. The biggest change (besides centralized offset management) is that
> each topic+partition is owned by a specific broker (the leader). When
> producing messages, you must send them to the correct leader. This requires
> that clients maintain some state of what belongs where which is a pain, but
> such is the cost of replication.
>> Thanks and cheers!
>> marc
>>  -David

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message